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ii Towards The National Assessment of Educational Progress (NAEP) as an Indicator of Academic Preparedness for College and Job Training 


Introduction 


For more than a decade, the National Assessment Governing Board has been conducting 
research to determine the potential of the National Assessment of Educational Progress (NAEP) 
at grade 12 to serve as an indicator of academic preparedness for college and job training. This 
report provides the rationale for pursuing this goal, the Phase | research results from more than 
30 studies conducted in connection with the 2009 administration of 12°" grade NAEP, and the 
implications for reporting the 2013 NAEP 12" grade results. 


Indicators of many kinds are used to monitor 
critical aspects of national life and inform public 
policy. These include economic indicators 

(e.g., gross domestic product), health indicators 
(e.g., cancer rates), and demographic indicators (e.g., 
population trends by race/ethnicity and gender). 


NAEP serves the public as a national and 

state indicator of education achievement at the 
elementary and secondary levels. NAEP reports 
achievement results for the nation, states, and 
demographic subgroups. NAEP does not report or 
provide individual student results. 


NAEP monitors student achievement at key points 
in the elementary/secondary progression: grades 
4,8, and 12. According to a policy statement of the 
National Assessment Governing Board (National 
Assessment Governing Board, 2013b): 


° The 4" grade is the point at which the 
foundations for further learning are expected 
to be in place (e.g., when “learning to read” 
becomes “reading to learn”) 


° The 8” grade is the typical transition point to 
high school 


° The 12" grade is the end of the K-12 education 
experience, the transition point for most 
students to postsecondary education, training, 
the military, and the work force. 


Towards The National Assessment of Educational Progress (NAEP) as an Indicator of Academic Preparedness for College and Job Training 


NAEP is the only source of nationally 
representative i” grade student achievement 
results. State tests of academic achievement are 
usually administered before ioe grade and are 
quite different across the country. Likewise, 
college admission tests like the ACT and SAT are 
generally taken before i grade by a self-selected 
sample and, therefore, are not representative of all 
i" graders. 


Consequently, NAEP is uniquely positioned to 
serve as an indicator of academic preparedness 
for college and job training at grade 12—the point 
that represents the end of mandatory schooling 
for most students and the start of postsecondary 
education and training for adult pursuits. 


A wide array of state and national leaders has 
embraced the goal that oo grade students 
graduate “college and career ready.” These include 
the leadership and members of the National 
Governors Association (NGA), the Council of 
Chief State School Officers (CCSSO), the Business 
Roundtable (BRT), the U.S. Chamber of Commerce 
(the Chamber), a task force on education reform 

of the Council on Foreign Relations, and state and 
national political leaders (Fields and Parsad, 2012, 


pp. 3-4). 


NAEP and Academic Preparedness 


a 


The Governing Board believes that NAEP 
reporting on the academic preparedness of i 
grade students would afford an invaluable public 
service: providing an indicator of the human 
capital potential of today’s and future generations 
of the nation’s population. 


The Board began this initiative in 2004, after 
receiving recommendations from a distinguished 
blue-ribbon panel that had examined whether 
NAEP should continue assessing at the i grade. 


The panel stated that “America needs to know how 
well prepared its high school seniors are... [only NAEP] 
can provide this information...and it is necessary for our 
nation’s continued well being that it be provided.” 

The panel recommended that NAEP continue 

to assess at grade 12 and that the io” grade 
assessment be transformed to measure readiness 
for college, job training, and the military’ (National 
Commission on NAEP 12'" Grade Assessment and 
Reporting, 2004, p. 2). 


In acting on the panel’s recommendations, the 
Governing Board changed the term “readiness” 

to “academic preparedness.” This is because 
“readiness” is broadly understood to include both 
academic preparedness and other characteristics 
needed for success in postsecondary education and 
training, such as habits of mind, time management, 
and persistence (Conley, 2007). NAEP does not 
measure such characteristics. Rather, NAEP 

is designed to measure academic knowledge 

and skills. 


To transform 12™ grade NAEP into an indicator of 
academic preparedness, the Governing Board took 
several significant steps. 


1. The Board determined that measuring 
academic preparedness for college and job 
training should be an intended purpose of 
i grade NAEP (see http://www.nagb.org/ 
content/nagb/assets/documents/policies/ 


resolution-on-preparedness.pdf). 


2. The Board contracted with Achieve, Inc. in 
2005 to review the NAEP 12" grade reading 
and mathematics assessment frameworks 
and identify where changes, if any, would be 
needed. Modest changes were recommended 
(Achieve, 2005; Achieve, 2006). 


3. Accordingly, the Board made changes to the 
frameworks to be used for the administration 
of the 12" grade assessments, scheduled 
for 2009 and 2013 (National Assessment 
Governing Board, 2008a; 2008b). 


4. In 2007, the Board assembled a team of noted 
psychometricians, industrial/organizational 
psychologists, and K-12 and postsecondary 
researchers to serve as a technical panel, 
advising on validity research to conduct. 


5. In 2008, the technical panel recommended a 
comprehensive program of research (National 
Assessment Governing Board, 2009a). The 
validity of statements about academic 
preparedness in NAEP reports would be 
affected by the degree to which the results 
were mutually confirming. 


1 Prepared for the military was later subsumed under “prepared for job training” by Board action. 


Figure 1 presents a model of the research relevant tests, (2) statistical linking studies 


program, with five types of research displayed, to compare performance on NAEP and other 
the interrelationships that would be examined, relevant tests by the same sample of students, 
and the potential meaning of the research (3) judgmental studies in which experts identify 
results in terms of the NAEP score scale. the point on the NAEP scale that represents 

“just academically prepared”, (4) a survey of 
The Board began contracting for the research postsecondary education institutions’ use of 
studies in 2008, in connection with the 2009 tests and cut scores for placement into first-year 
administration of the 12" grade reading courses, and (5) benchmarking studies in which 
and mathematics assessments. More than 30 target groups of interest take NAEP (e.g., military 
research studies were completed during the recruits, freshmen college students, trainees in 
period 2009-2012 (see technical report at selected technical/career programs, etc.). 


http://www.nagb.org/what-we-do/ 
preparedness-research.htm1). The Research Findings 
The research findings were generally consistent across 


The research studies were of five types: studies. Brief examples of the research results follow, 
(1) content alignment studies to compare the which include data from different years (e.g., see 
content of NAEP with the content of other Figures 2 and 3). 


Figure 1: Model of the Preparedness Research Program 
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The content of the 12™ grade NAEP reading and 
mathematics assessments was found to be similar to 
widely recognized tests used for college admission 
and placement (see http://www.nagb.org /what- 


we-do/preparedness-research/types-of-research/ 


content-alignment.html). 


Performance by the same sample of students on 
NAEP and the SAT mathematics and reading 
tests was correlated at 0.91 and 0.74, respectively 


(see http://www.nagb.org /content/nagb/assets/ 
documents/what-we-do/preparedness-research/ 
statistical-relationships/SAT-NAEP Linking _ 


Study.pdf). 


Statistical linking studies examined performance 
on NAEP in relation to the college readiness 
benchmarks adopted by the College Board for the 
SAT reading and mathematics tests. The average 
NAEP score of students scoring at the SAT college 
readiness benchmarks was 163 in mathematics and 
301 in reading: These results were confirmed by 
other studies that examined performance on NAEP 
in relation to the college readiness benchmarks 
adopted for the ACT (see Figures 2 and 3)° 


A longitudinal study followed a representative 
sample of Florida i grade NAFP test-takers 
into the state’s public colleges. The longitudinal 
study permitted an analysis of performance on 
NAEP and actual student outcomes. In the first 
year of this study, an analysis was conducted 

of performance on NAEP and (1) enrollment in 
regular versus remedial courses, and (2) first-year 
overall college grade point average (GPA). The 
average NAEP score of the students who were not 
placed into remedial courses or who had a first- 


year college GPA of B- or better was similar to, and 
thus confirmed, the NAEP to SAT linking studies 
cited previously (see Figures 2 and 3). 


Results from the more than 30 studies were used to 
develop a validity argument to support proposed 
inferences (claims) about academic preparedness 
for college in relation to performance on ‘Pig grade 
NAEP. The validity argument was reviewed by two 
independent technical reviewers. The technical 
reviewers concluded that the validity argument 
supports the proposed inferences. 


The complete research reports and the validity 


argument can be found at http://www.nagb.org/ 
what-we-do/preparedness-research.html. The two 


independent reviews can be found in Appendix A 
and Appendix B of the validity argument at http:// 
www.nagb.org/content/nagb/assets/documents/ 


what-we-do/preparedness-research/nagb-validity- 
report.pdf. 


The research results support inferences about 
NAEP performance and academic preparedness 
for college at the national level. The research 
results do not support inferences about NAEP 
performance and academic preparedness for job 
training. A second phase of NAEP preparedness 
research was begun using NAEP data collected 
in 2013. The results from the second phase of 
research will be examined to determine the extent 
to which they confirm existing results. They also 
will be examined to determine their support for 
(1) inferences about academic preparedness 

for college at the state level, (2) reporting by 
demographic subgroups, and (3) inferences about 
academic preparedness for job training. 


2 The average score of 301 in reading is not statistically different than the cut score for the Proficient achievement level on NAEP, which is 302. 
Consequently, the Governing Board developed the inference for reading in relation to the score of 302. 


3 The respective SAT and ACT college readiness benchmarks in effect in 2009 were used for the analyses in the studies displayed in 


Figures 2 and 3. 


Figure 2: NAEP 12'" Grade Academic Preparedness Research: Reading 


Average Scores and Interquartile Ranges on NAEP for Selected Variables and SAT/ACT College Readiness Benchmarks, 
from the 2009 NAEP SAT Linking Study and 2009 Florida Longitudinal Study 
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SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, previously unpublished analyses, March 2013; National Assessment 
Governing Board, Statistical Linking of National Results from NAEP and SAT (2009), previously unpublished analyses, March 2013; National Assessment Governing Board, Longitudinal 
Statistical Relationships for Florida NAEP Examinees: First-Year College Performance Outcomes (2009-2010), previously unpublished analyses, March 2013. 


Figure 3: NAEP 12" Grade Academic Preparedness Research: Mathematics 


Average Scores and Interquartile Ranges on NAEP for Selected Variables and SAT/ACT College Readiness Benchmarks, from the 2009 
NAEP/SAT Linking Study, 2005 High School Transcript Study, 2009 High School Transcript Study, and 2009 Florida Longitudinal Study 
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SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, NAEP High School Transcript Study (2005, 2009), previously 
unpublished analyses, March 2013; National Assessment Governing Board, Statistical Linking of National Results from NAEP and SAT (2009), previously unpublished analyses, 
March 2013; National Assessment Governing Board, Longitudinal Statistical Relationships for Florida NAEP Examinees: First-Year College Performance Outcomes (2009-2010), 
previously unpublished analyses, March 2013. 


A Transition to Reporting 
on Academic Preparedness 


‘ 


The reporting of the 12" grade results for 2013 
represents a transition point for NAEP. 


The interpretations of the 2013 NAEP ea grade 
reading and mathematics results related to 
academic preparedness for college set forth in this 
report are foundational, a first step, and subject 

to adjustment in the future. Accordingly, the 
percentages of students at or above the NAEP 
scores described in the inferences for reading 
and mathematics below should be considered 


provisional estimates. 


The technical reviewers concluded that these 
interpretations are supportable. They are included 
in this report to promote public discussion about 
their meaningfulness and utility. 


The Context for Academic 

Preparedness for College 

In the United States in 2014, there is no single, 
agreed upon definition of “academic preparedness 
for college.” Postsecondary education in the U.S. is 
a complex mix of institutions, public and private, 
that have different admission requirements and 
different procedures and criteria for placing 
individual students into education programs. 


In this complex mix are 2-year institutions, 4-year 
public and private institutions with a wide range 
of selectivity, and proprietary schools. Institutions 
range from highly selective (i.e., with admission 
criteria including very high grade point averages, 
successful completion of rigorous high school 
coursework, and very high SAT and/or ACT 
scores) to open admission (i.e., all applicants 

are admitted). 


Even within institutions, requirements may 

vary across majors or programs of study. For 
example, the mathematics and science high school 
coursework and academic achievement needed 

for acceptance into an engineering program in a 
postsecondary institution may be more rigorous 
than the general requirements for admission to the 
institution or for a degree in elementary education 
in that institution. 


The need for clarity and coherence in describing to 
students, parents, and educators what constitutes 
academic preparedness for college is widely 
recognized. State education agencies responsible 
for K-12 and postsecondary education have 

been working for several years—collectively as 

a part of consortia collaborating with national 
organizations or individually for their own state— 
on standards and tests to define and measure 
academic preparedness for college. This work is 
expected to continue for the next several years. 


In addition, the SAT and the ACT college 
admission testing programs continue to be 
refined, in terms of test content and research on 


indicators of college readiness. 


An Evolving Definition of Academic 
Preparedness for College for NAEP 

Given the diversity of postsecondary education 
institutions, it is essential to provide a reasonable 
definition of academic preparedness for NAEP 
reporting. The definition should be relevant to 
NAEP’s purpose of providing group estimates of 
achievement. (It is important to note that NAEP 
does not provide individual student results.) 


The definition should be meaningful to NAEP’s 
primary audiences: the general public and national 
and state policymakers. 


In order to design the NAEP ea grade 
preparedness research, a working definition 

of preparedness was needed. As a working 
definition, it would change and evolve based on 
the research results. The working definition was 
intended to apply to the typical degree-seeking 
entry-level student at the typical college. For the 
working definition, “academically prepared for 
college” refers to the reading and mathematics 
knowledge and skills needed for placement into 
entry-level, credit-bearing, non-remedial courses 
in broad access 4-year institutions and, for 2-year 
institutions, the general policies for entry-level 
placement, without remediation, into degree- 
bearing programs designed to transfer to 4-year 


institutions. 


It is important to note the focus on “placement” 
rather than “admission.” This distinction was 
made because students who need remedial 

courses in reading, mathematics, or writing may 
be admitted to college, but not placed into regular, 
credit-bearing courses. The criterion of importance 
is qualifying for regular credit-bearing courses, not 
admission. 


The working definition was not intended to reflect 


* academic requirements for highly selective 
postsecondary institutions; 


¢ the additional academic requirements for 
specific majors or pre-professional programs, 
such as mathematics, engineering, or medicine; 


or 


* academic requirements applicable to entry into 
certificate or diploma programs for job training 
or professional development in postsecondary 


institutions. 


The working definition was focused on the 

first year of college; it did not address college 
persistence beyond the first year or completion 
of a degree. The definition was intended to apply 
in general across a broad range of programs and 
majors, but not specifically to any particular 


program or major. 


Inferences for NAEP Reporting 

The NAEP preparedness research does not affect 
the 2013 NAEP 12" grade results in any way. The 
distribution of student achievement is unchanged. 
That is, the average scores, the percentiles, and the 
achievement level results are not impacted by the 
NAEP preparedness research. 


The independent technical reviewers confirmed 
that the research findings support inferences 
about performance on NAEP io grade results in 
reading and mathematics in relation to academic 
preparedness for college. The research findings 
had an impact on the working definition of 
academic preparedness for college that was used 
for the purpose of designing the preparedness 
research. In contrast to the working definition’s 
focus on being just “minimally academically 
prepared” (i.e., not needing remedial reading 

or mathematic courses in college), the research 
findings point more to a level of academic 
achievement consistent with attainment of a first- 
year overall college grade-point average of B- or 
better and a low probability of placement into 


remedial courses in college. 


Inference for Reading 

In the NAEP/SAT linking study for reading 
(Figure 2), the average NAEP score for a grade 
students scoring at the SAT college readiness 
benchmark for critical reading was 301, not 
significantly different statistically from the 
Proficient cut score of 302. The results from the 
Florida longitudinal study were confirmatory. 


These data, together with the content analyses 

that found NAEP reading content to be similar to 
college admission and placement tests, support the 
inference for reading that 


Given the design, content, and characteristics of 
the NAEP 12" grade reading assessment, and the 
strength of relationships between NAEP scores and 
NAEP content to other relevant measures of college 
academic preparedness: 


the percentage of students scoring at or above 
a score of 302 on the Grade 12 NAEP scale in 
reading is a plausible estimate of the percentage 
of students who possess the knowledge, skills, 
and abilities in reading that would make them 
academically prepared for college. 


A score of 302 corresponds to the cut score for the 
Proficient achievement level in 12" grade reading. 


In 2013, 38% of 12" graders nationally scored at or 
above 302 in reading. 


Inference for Mathematics 

In the NAEP/SAT linking study for mathematics 
(Figure 3), the average NAEP score for i” grade 
students scoring at the SAT college readiness 
benchmark for mathematics was 163, lower than 
and significantly different from the Proficient cut 
score of 176. The results from the High School 
Transcript Study and the Florida longitudinal study 
are confirmatory. 


These data, together with the content analyses that 
found NAEP mathematics content to be similar to 
college admission and placement tests, support the 
inference for mathematics that 


Given the design, content, and characteristics of the 
NAEP 12" grade mathematics assessment, and the 
strength of relationships between NAEP scores and 
NAEP content to other relevant measures of college 
academic preparedness, 


the percentage of students scoring at or above 
a score of 163 on the Grade 12 NAEP scale 

in mathematics is a plausible estimate of the 
percentage of students who possess the 
knowledge, skills, and abilities in mathematics 
that would make them academically prepared 
for college. 


A score of 163 in mathematics is between the cut 
scores for the Basic and Proficient achievement 
levels in 12" grade mathematics. 


In 2013, 39% of 12" graders nationally scored at or 
above 163 in mathematics. 


The research results for mathematics support 

a related inference—that students in the 
distribution at or above the NAEP Proficient level 
in mathematics are likely to be academically 
prepared for college. However, the percentage of 
such students would be substantially less than the 
percentage in the distribution at or above 163, and 
thus, would underestimate the percentage of ie 
grade students in the U.S. who are academically 
prepared for college. 


To consider the plausibility of these estimates, 
comparisons can be made with the percentages 
of students who met the ACT or SAT college 
readiness benchmarks. These data, displayed in 
Table 1, are from reports published by ACT and 
the College Board, respectively. 


The table presents information about students 
who were seniors in 2009 (ACT) and in 2010 (SAT). 
Thus, the ACT data are for the same student 
cohort as the NAEP research data, but the SAT 
data are for a cohort that followed one year later. 


It also must be noted that, unlike the NAEP 
results, neither the ACT nor the SAT results 
represent all i” graders. Further, there is overlap 
among ACT and SAT test-takers, with about 

20% estimated to take both tests. 


Assuming that a substantial portion of students 
who do not take either test are not academically 
prepared for college, it is not inconsistent that the 
NAEP percentages displayed in the table are lower 
than those for the respective college readiness 
benchmarks. 


Table 1 


Percentages* scoring at/above 302 in Reading 
on NAEP and at/above 163 in Mathematics 

on NAEP, and at/above ACT and SAT College 
Readiness Benchmarks 


Reading Mathematics 
ACT (2009) 53 42 
SAT (2010) 50 54 
NAEP (2009) 38 40 


* About 48% of 12" graders took the ACT or SAT. NAEP is intended to 
represent 100% of 42" graders. 


Source: ACT (http://www.act.org/newsroom/data/2009/benchmarks. 
html); SAT (Wyatt et. al. (2011); National Center for Education 
Statistics.) 


Limitations on Interpretation 

and Other Caveats 

False Negatives and False Positives 

Some proportion of Pag grade students scoring 
below 302 on the 12" grade NAEP Reading or 
below 163 on the Mathematics Assessment are 


e likely to be academically prepared for college 
¢ not likely to need remedial/developmental 


courses in reading or mathematics in college, 


but with a lower probability than those at or above 
302 in reading or 163 in mathematics. 


In addition, some proportion of io” grade 
students scoring at or above 302 on the a” 
grade NAEP Reading or 163 on the Mathematics 
Assessment may 


¢ not be academically prepared for college 
¢ need remedial/developmental courses in 


reading or mathematics in college. 


Academic Preparedness Indicators 

and Inferences 

The NAEP reading and mathematics scores 
identified in this report are intended to serve 

as preliminary indicators of the academic 
preparedness of the U.S. students. They are not 
intended to represent or be used as standards 

for minimal academic preparedness for college. 
The use of these indicators is intended solely to 
add meaning to interpretations of the ag grade 
NAEP reading and mathematics results in NAEP 
reports. It is important to note that the measures 
and outcomes used in the validity argument to 
support the inferences in this report are at a level 
that is somewhat beyond a “minimally acceptable” 


10 


or “just qualified” level for placement into entry- 
level, credit-bearing courses. Thus, the average 
NAEP scores associated with these indicators 
are based on measures and outcomes that point 
more to solid academic achievement than to a 


“minimal” level of academic preparedness. 


GPA of B- or Better 

The variable “first-year GPA of B- or better” was 
selected because of its use as a research-based 
criterion in defining college readiness benchmarks 
developed by an acknowledged leader in college 
testing programs—the College Board. The College 
Board had agreed to partner with the Governing 
Board in the study linking performance on 

12" grade NAEP with the SAT. Another leader 

in college testing programs, ACT, Inc., has 
developed similar benchmarks for its college 
admission assessments using a similar criterion 
and similar methodology. Because they are based 
on research related to college outcomes, and 
because performance on the respective tests could 
be linked to performance on NAEP, the college 
readiness benchmarks used by these testing 
programs were relevant, useful points of reference 


for the NAEP preparedness research. 


The College Board set a score of 500 on the 

SAT Mathematics and Critical Reading tests 

as its college readiness benchmarks in those 
areas. Based on its research, the College Board 
determined that the score of 500 predicts, with a 
probability of .65, attainment of a first-year overall 
GPA of B- or higher. Similarly, the ACT college 
readiness benchmarks are based on research 
indicating a .50 probability of attaining first-year 
grades in relevant courses (e.g., college algebra and 
courses requiring college-level reading) of B or 
better and .75 probability of C or better. 


The inferences are not intended to convey that 
a B- or any particular grade should be deemed 
a standard or goal for postsecondary student 
outcomes. This criterion was selected to foster 
comparability across the preparedness research 
studies, where applicable. However, it does seem 
self-evident that achieving a first-year GPA of 
B- or better, without enrollment in remedial/ 
developmental courses, lends support to the 
likelihood of having possessed academic 
preparedness for first-year college courses upon 
entry to college. 


Data Limitations 

The NAEP preparedness research studies are 
comprehensive and the results consistent and 
mutually confirming. However, for reading, 
the statistical studies are limited to one year for 
data at the national level and to one state-based 
longitudinal study. For mathematics, there are 
two separate years of data at the national level 
and one state-based longitudinal study (see 
Figure 3). Therefore, more evidence exists to 
support the plausibility of inferences related to 
mathematics than to reading. Further, at this 
time, inferences will be made at the national 
level only. Research completed to date does 

not support inferences at the state level or for 
student subgroups. 


Preparedness for Job Training 

The research to date with respect to academic 
preparedness for job training does not support 
conclusions relative to the NAEP scale. Options 
for additional research on this topic will 
continue to be reviewed and discussed by the 
Governing Board. 


Conclusion 


The independent technical reviewers found the 
Governing Board’s preparedness research to be 
methodical, rigorous, and comprehensive. They 
concluded that the research findings support 
the use of inferences in NAEP reports about ea 
graders’ academic preparedness for college. 


The interpretations of NAFP results in relation 
to academic preparedness for college are being 
reported on a preliminary basis. They are 
provided to help foster public understanding 
and policy discussions about defining, 


measuring, validating and reporting on academic 


preparedness for college, both for NAEP and for 
related initiatives more broadly. 


Including these inferences in NAEP Pe grade 
reports is intended to add meaning to the 
interpretation of the NAEP i? grade results. 


However, the potential for misinterpretation 
exists. For these reasons, the section above on 
limitations on interpretation and other caveats is 


included in this report. 


The Governing Board will monitor the use 

of these inferences, as well as unintended 
consequences arising from their use, as a part of 
the next phase of the preparedness research. The 
Board will take appropriate steps to clarify any 
misconceptions that may be discovered through 


this monitoring. 


The next phase of the preparedness research 

is being conducted in connection with the 
NAEP reading and mathematics assessments 
administered in 2013. The research will be used 
to refine and possibly expand reporting on 12" 
grade academic preparedness. 
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